Facial Displays , Emotional Expressions and Conversational Acts

“Emotional expression is multifaceted – expression is determined both by a person’s reaction to an event and by the attempt to manipulate this expression for strategic reasons in social interaction.” (Scherer, 2001). In this paper we present some thoughts on the relation between emotion, facial expression and dialogue that have lead us to develop a model of dialogue in which the social views on language use are hooked up with theories of the sociology of emotions and in which facial expressions are mainly studied as discourseoriented actions without loosing track of the emotional antecedents and consequences that gave rise to them or that they raise. 1 Emotions-Faces-Dialogue Traditional spoken dialogue systems abstract away from many processes found in natural, face-to-face conversations. The dialogues with these systems are turn-based, task-oriented and make use of limited input and output modalities. The work on Embodied Conversational Agents has been trying to move away from these limitation in several ways. First, by extending the communicational signaling to other modalities taking into account facial expressions, gestures and also posture and gaze, for instance. Secondly, by not only taking a task-centered approach to conversation but by also paying attention to modeling the emotion processes, the personality of the conversational agent, and the social context of interaction. A third theme that has received some attention as well is the move away from turn-based systems towards continuous interaction. All this has lead to systems in which more and more is modelled of the intricacies of natural conversations and the various ways that conversations involve the participants in this activity. In this paper we present some thoughts on the relation between emotion, facial expression and dialogue that have lead us to develop a model of dialogue in This work was partly supported by the European Union 6 FWP IST Integrated Project AMI (Augmented Multiparty Interaction, FP6-506811, publication AMI-157). which the social views on language use are hooked up with theories of the sociology of emotions and in which facial expressions are mainly studied as discourse-oriented actions. 2 Faces and Emotions Ekman’s views on faces and emotions Facial expressions have been taken to constitute both antecedents as in the case of the facial feedback hypothesis and consequences of emotions. The subject of emotions and facial expressions has been amply discussed in the literature and thanks to Paul Ekman and John Cleese every person in the world now thinks that there are six basic emotions associated with six universal facial expressions. We start the discussion on faces and emotions with expounding Ekman’s views not because we necessarily endorse them, but because we want to introduce a number of important aspects of the relations between emotions and faces. In Telling Lies, Paul Ekman (2001, pages 124-125), puts it like this. The involuntary facial expressions of emotion are the product of evolution. Many human expressions are the same as those seen on the faces of other primates. Some of the facial expressions of emotion – at least those indicating happiness, fear, anger, disgust, sadness, and distress, and perhaps other emotions – are universal, the same for all people regardless of age, sex, race, or culture. These facial expressions are the richest source of information about emotions, revealing subtle nuances in momentary feelings. [...] The face can show: • which emotion is felt – anger, fear, sadness, disgust, distress, happiness, contentment, excitement, surprise, and contempt can all be conveyed by distinctive expressions; • whether two emotions are blended together – often two emotions are felt and the face registers elements of each; • the strength of the felt emotion – each emotion can vary in intensity, from annoyance to rage, apprehension to terror, etc. What Ekman is not saying though is that: • The face is only an involuntary emotional signal system. • All facial expressions of emotion are universal. • Each emotion is always expressed in the same way. • Only basic emotions are displayed on the face. Facial Displays, Emotional Expressions and Conversational Acts • The face only displays emotions. In fact, on the pages that follow he writes: There is not one expression for each emotion, but dozens and, for some emotions, hundreds of expressions. Every emotion has a family of expressions, each visibly different one from another. This shouldn’t be surprising. There isn’t one feeling or experience for each emotion, but a family of experiences. [...] Already we have evidence that there are more different facial expressions than there are different single words for any emotion. And he continues: There are thousands of facial expressions, each different from one another. Many of them have nothing to do with emotions. Many expressions are what we call conversational signals, which, like bodymovement illustrators, emphasize speech or provide syntax (such as facial question marks or exclamation points). There are also a number of facial emblems: the one-eye closure wink, the raised eye-browsdroopy upper eyelid-horseshoe mouth shrug, the one-eyebrow-raised skepticism, to mention a few. There are facial manipulators, such a lip-biting, lip sucking, lip wiping, and cheek puffing. And then there are the emotional expressions, the true ones and the false. Furthermore, Ekman writes: But as I said, the face is not just an involuntary emotional signal system. Within the first years of life children learn to control some of these facial expressions, concealing true feelings and falsifying expressions of emotions not felt. Parents teach their children to control their expressions [...]. As they grow up people learn display rules so well that they become deeply ingrained habits. After a time many display rules for the management of emotional expression come to operate automatically, modulating expression without choice or even awareness. [...] I believe that those habits involving the management of emotion – display rules – may be the most difficult of all to break. (Ekman, 2001, page 125). Researchers have taken issue with many of the positions held or supposedly held by Ekman on the relation between facial expressions and emotions. These critiques involve, amongst others, the issue of universality of certain expressions, the idea that categorical emotions should be associated with a specific facial configuration. For instance, some would claim that it is more fruitful to look at the determinants of facial expressions in terms of a dimensional approach (Russell, 1997), appraisal checks (Scherer] or action readiness (Frijda & Tcherkassof, 1997) instead of categorical emotions. Instead of linking particular emotions or emotion components to a complete facial expression, Smith and Scott (1997) take a componential view: linking emotion dimensions to components of the facial expression (see also Kaiser & Wehrle, 2001). Also the idea that emotion automatically leads to a particular expression, the notion of display rule, the evolutionary views on facial expressivity, and the role of social context are debated in the literature. For our research the latter question is the most important as we approach the data from a conversational analytic rather than an emotion-theoretic point of view. Social context The above quote by Ekman in which he talks about display rules, presents the outline of the general model of his neuro-cultural model. According to Fridlund (1994), Ekman is a typical advocate of an “Emotions View”: “they share the belief in the centrality of emotion in explaining facial movements.” The most frequent Emotions View of faces is, as I termed it elsewhere, essentially a “two-factor” model that posits two basic kinds of faces. First are the innate reflex-like faces that read out ongoing emotion; these are termed “facial expressions of emotion”. Second are learned, instrumental faces that connote emotion that is not occurring; these reflect everyday social dissimulation such as the smile of politeness. The facial expressions observed in everyday life represent an interaction of emotional instigation and cultural adulteration. In Fridlund’s behavioural ecology view of faces, facial displays are communicative acts serving social motives. The primary function is not to “express” internal states but to “signal” one’s intention to others. According to several researchers, it is, however, not at all obvious that these views should be seen as completely incompatible (see Jacobs et al, 1999, for instance). As such, this debate between Ekman and Fridlund on the general model is of minor importance for the questions that we are concerned with. What is important, though, is the fact that the social context is a crucial factor with respect to facial expressivity. This is not only clear from the well-known effects of social context on the expressions that appear on faces (see: Kraut & Johnston, Young and Fry 1966, Chapman and Wright (1976); but also by Ekman studies, themselves. For him, in trying to show the connection between the universal expressions of the basic emotions, he has to take care to exclude the social influence as much as possible. The least one could say is that “The presence of another person produces variation in facial behaviour” (Wagner & Lee, 1999) and Friesen and Ekman’s Japanese-American study is a case in point (Friesen, 1972). Our research does not start from a question like “What’s the relation between facial expression and emotion?” But rather from a question like “How do facial expressions contribute to the communication in conversational settings?” The literature on facial expressions and emotions is not without relevance for this research question but has to approached with some care. [M]ost studies of facial expressions have used highly constrained laboratory situations [...] These studies have been very informative about a number of theoretical issues that are relevant to noninteracting persons. However, there has been a tendency, explicit or implicit, to generalize the results of such research to less constrained, more social situations. This has led to the general assumption that the main function of facial expressions in social, interactive situations is to express emotional feelings. We have argued that, in order to understand nonverbal behaviour in social situations, it is necessary to study that behaviour together with its verbal context. As we have shown, such an integrated approach can lead to different interpretations of the functions of facial expressions. This kind of approach is essential if we are to study communicative behavior in social situations. (Wagner and Lee, 1999) 3 Emotions and Conversations Interestingly, in his search for universal expressions of basic emotions, Ekman takes care to consider precisely those circumstances in which truly felt emotions are expressed without consideration of display rules. This means that in many experiments a situation is created in which the subjects are alone and care is taken that it is not obvious to them that they are observed. For a conversational agent, these findings are not very useful as the purpose of the agent, by definition, is to interact. So, if we make our agents more “emotional”, this does not entail that we should have their emotions expressed through their facial displays. Conversations are social encounters in which agents, like people, should not be expected to show how they (really) feel without further ado. In terms of Ekman’s views on facial expressions and emotions, the research on facial expressions for embodied conversational agents should not focus on the voluntary, universal expression of basic emotions but concentrate on the identification of the kinds of display rules that are appropriate for the nature of the agent, its character, culture and the kind of encounter: the nature of the exchange and the nature of the interlocutor. For an engineer building synthetic faces to be used in conversational settings the research program as formulated by Bavelas and Chovil (1997) as below appears to be more to the point. The research [...] represents the first stage in a program of study to investigate facial displays as discourse-oriented actions. In this research, facial displays are regarded as linguistic elements of a message rather than outputs or “spillover” of emotion processes. In this way of approaching the subject of facial expressions the central point of attention is the social setting in which the conversation proceeds. Chovil (1991) writes: Although facial displays are undoubtedly used at times to convey information about how a person is feeling or reacting, emotion displays do not account for the majority of displays that occur. Ekman and Friesen [...] found that in nearly 6,000 facial displays of psychiatric patients, less than one-third were classifiable as facial expressions of emotions. This suggests that, although some facial displays may convey information about emotion, there are a substantial number of displays that we know very little about. Based on her own research Chovil (1991) concludes that hardly 20% of the displays in face-to-face conversations are affective. However, this does not mean that emotions no longer have a role to play in generating the facial expressions of talking heads. Discourseoriented actions should not be construed too narrowly as being related merely to the linguistic exchange; which would mean looking specifically at conversational signals. Rather, one should look at conversation as a form of social interaction and when one considers emotions this could be by looking at t he role of facial expressions in processes such as impression management, facework (Goffman, 1959), emotion work and emotion management (Hochschild, 1979, Thoits 1989), reflexivity (Rosenberg, 1990), empathy (Clark, 1997), the conception of self and identity through interpersonal interaction (Mead 1936, Shott, 1979, Stryker 2004), and affect control (Heise, 1989). As far as affect influences the choice of expression, it should be realized that in a conversation the affective state is co-determined mainly by the events that happen during the conversation. This involves, on the one hand, the “business” goals of the conversation and how one succeeds with those play a role, but also the social rapport one is trying to build up. how well one likes the interlocutor, how one wants to present oneself in the conversation and how well one succeeds, etcetera). Argyle (1993) argued that interpersonal functions of facial displays showing how one feels and thinks, about the other and the relation to self are very important in conversations. Many of the kinds of facial expressions found in conversations relate to this aspect of interaction. For instance, the cases of portrayal, identified by Chovil or the phenomenon of mimicry in which listeners “enact” (out of sympathy or whatever) the feelings of the person who is talking to them or the feelings of the character talked about (Bavelas et al. 1986). 4 Faces and Conversations When one considers the case of facial expressions in conversations one should thus realize that they can serve several functions. These have to do with the many different kinds of goals and levels of actions that are going on in a conversation at the same time. Different patterns of gaze, for instance, has been analyzed as playing a role in indicating addresseehood, effecting turn transitions, as displays of attentiveness. Gaze may also reflect the social status. One may look away to avoid distraction and to concentrate, but also to indicate one doesn't want to be interrupted. One looks to the other in order to get cues about mood and disposition of the other, to establish or maintain social contact. Gazing away may also reflect hesitation, embarrassment or shyness. There are three important points to note about this list that tell us more about the nature of conversation and the role of facial expressions. First of all, it shows the fact that conversation involves many levels. There is the actual act of saying something and attending, the act of meaning something by what is being said and understanding this on the part of the listener, and the act of getting the listener do something: believe, feel, commit, etcetera. There are actions involved with the actual business of the conversation, but also actions that are directed at managing the conversation the conversation (again on all kinds of levels: ensuring attention, ensuring understanding, ensuring the uptake of the proposed projects and ensuring the right kind of social bond, attitude and feeling). Second it shows that behaviour “means” in different ways. A behaviour may act as an involuntary cue or as a conventionalized symbol. Finally, all of the actions are directed at the other. They are meant to be attended to, and to be understood by the interlocutor. The social nature also shines through on the emotional level where gaze and many other expressions affect the social emotions, such as embarrassment. In earlier work we have tried to incorporate the socialaffective, interpersonal level into our dialogue systems (Heylen et al. 2004). We have now started to look in more detail at facial expressions in conversations much as we look at other conversation actions along the lines of the program sketched by Bavelas and Chovil (1997) one could say. But, whereas their work is mainly concerned with categorizing the facial displays in terms of the types of functions they serve in a conversation, we are concerned more specifically with the specific act they serve. Conversations are interactive processes in which our actions are directed at persons for communicative purposes. Many of our expressions are therefore consciously produced or controlled to inform the other persons about things that we really want the other person to know. We do not simply “reveal” our mental state, but we choose expressions in the same way that we use natural language expressions. Many facial expressions could be said to functions as part of a “speech act”. Petukhova (2005) looked at a number of meetings, that she had annotated with a refined dialogue act scheme proposed by Bunt (2000). This scheme takes into account many levels of the conversational organization, including some social aspects. Also, the meetings are being annotated on the affective dimensions. Next, she collected a database of clips from the meeting which showed the kinds of nonverbal behaviours displayed with each of the functions. Table 1, shows the major behaviours that are associated with several kinds of communicative functions. The functions listed in this table are the so-called communication management functions. Similar tables were constructed for topic, contact and turn management and for feedback functions. When looking at the facial displays, one can see that typical expressions for emotions and mental states, like “surprise”, “puzzled”, “guilty” can be associated with specific communicative acts. In certain cases the communicative act is only expressed by the nonverbal act in context, in other cases, the nonverbal act, only accompanies Communicative act Face Posture Hand/Arm gestures Head Pause neutral Neutral Gesturing stops Stalling Thinking